Purpose

The purpose of the present document is to provide an installation guide and to describe the mode of operation of the SPICE Archive Generation pipeline (ARCGEN) and a summary of its main functionalities. After reading this document the user should be able to install, operate with and fully understand the software package.

Running ARCGEN in a nutshell

1. Obtain ARCGEN from BitBucket: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/arcgen/

2. Install from top level directory with "pip3 install -e ."

3. Generate an ARCGEN configuration file as from arcgen/arcgen/test/exomarsrsp.json

4. Generate the appropriate directory structure as derived from the configuration file

5. Obtain and install your SPICE Kernel Dataset: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS

6. Run with "arcgen yourconfig.json"

Overview

Auxiliary data are those that help scientists and engineers to determine the location and orientation of the spacecraft, when and how an instrument was acquiring scientific data. These data also help to determine what other relevant events were occurring on the spacecraft or ground that might affect the interpretation of the scientific observation or the S/C systems performance. Software applications are required to know what were the location, size, shape and orientation of the observed target in addition to these auxiliary data. Almost all NASA and ESA planetary missions have embraced the use of the SPICE system for ancillary data archiving and for science data analysis. Although the Flight Dynamics Division provides software to read the position and orientation files of the orbiters, most of the Principal Investigators (PI) have pointed out their interest in having all the auxiliary data distributed in SPICE format.

The ESA SPICE Service (ESS) of the Cross Mission Support Office is responsible for the generation of the PDS3 and PDS4 SPICE Archives for the applicable ESA missions. ARCGEN supports the ESS engineers and selected users such as the ExoMarsRSP ROCC to generate these PDS3 data sets and PDS4 bundles.

Applicable Documents

Reference Documents

Hardware and Software requirements

ARCGEN is a Python 3.6.x package that uses a set of standard Python packages and the SpiceyPy package.

ARCGEN is designed to work with UNIX systems namely with Linux and with MacOS. The hardware requirements are minimal and any modern computer/server will suffice. The software is able to run for example in a minimal CentOS 7 Virtual Machine:

• Architecture: x86_64 (64 bits)

• Memory: 1GB/logical CPU

• Disk space: 10GB

Nevertheless it is recommended to allocate more resources for ARCGEN to run appropriately, the "Regression Tests" section includes further information on resource allocation and expected performances.

The software requirements are the following:

• Python 3.6 or higher

• Python packages:

  • numpy >= 1.8.0

  • spiceypy

• Git (possibly, for the obtention of SPICE Kernel Datasets)

Obtaining the Software

Configuration

The next step to use ARCGEN is to configure it.

The configuration has to be edited and prepared by the user. It consists of a JSON file and an example from the test directory of ARCGEN can be used as baseline (arcgen/arcgen/test/exomarsrsp.json). Any name can be provided to the configuration file, the file extension has to be *.json. The configuration file will determine the directory structure that you need to setup for ARCGEN. The recommendation is to create a directory out of the ARCGEN directory structure and to create one per intended ARCGEN execution/project.

The following table describes all the available configuration items (please note that all the items are strings):

Once the configuration file has been generated it can be placed anywhere in your computer, it is recommended to put it in the same directory level as where you will have the directory structure as a result of the ARCGEN execution. What follows is an example of an ARCGEN configuration file:

If we consider a Mars-Express configuration file, after an execution we could have a directory structure as follows:

Validating PDS3 Data Sets

It is also a good practice to run additional checks on the generated archive to make sure the files have been generated correctly, the format is adequate and the PDS standards have been followed properly. In the directory arcgen/arcgen/utils, the python script checkText.py can be used to check some PDS3 format consistencies, end of lines, line lengths and label keywords.

In addition the legacy PSA tool DVal tool may be used as well to perform some checks on the Data Set. Download the Data Validator command line tool from ftp://ssols01.esac.esa.int/pub/software/DValNG/index.html#advanced. Unpack the gzipped tar file. Executables are in DatasetValidator/bin. Before running any checks, run ConfigSynchronizer. Calling GrammarChecker without arguments lists possible options and parameters. In order to create a log file listing the issues found by the tool, GrammarChecker as:

It will find issues related to incorrect line endings, non ASCII characters or incorrect label structure.

For more information on how to validate a PDS3 data set contact the ESS.

Validating PDS4 Bundles

For PDS4 Bundles running the PDS Validate Tool is recommended [RD-VAL].

The context products required for each mission are available in the arcgen/arcgen/config directory. An example of how to execute the validate tool is available at arcgen/arcgen/tests/test_exomarsrsp_validate.sh

For more information on how to validate a PDS4 bundle contact the ESS.

ARCGEN Python packages

ARCGEN does not have any non-Python internal executables, instead, ARCGEN makes use of the non-standard Python package SpiceyPy [RD-SPY].

ARCGEN exit codes

When a command finishes execution, it returns an exit code. The exit code is not displayed on the command line screen by default. To examine the exit code, you need to examine a special variable, "$?", that can be displayed with the command "echo $?". In bash in general the exit status is 0 if the command succeeded, and 1 if failed. The exit codes for ARCGEN are as follows:

• Run successfully: 0

• Dry-run (without input): 0

• Display usage 'adcsng -h': 0

• Run with errors: 1

• Executed with wrong arguments: 2

Regression tests content

Currently there are only three tests defined for ARCGEN. More tests shall be implemented.

• test_em16_v001: This test validates the first PDS4 bundle generated by ARCGEN for ExoMars 2016

• test_em16_v002: This test validates the first PDS4 bundle increment done by ARCGEN for ExoMars 2016

• test_vex_v002: This test validates the first PDS3 dataset generated by ARCGEN for Venus-Express. Note that name is .._v002 but this is just because it uses as reference the version 2 of the VEX dataset (VEX-E-V-SPICE-6-V2.0), it's not a dataset increment.

Each of these tests are implemented in a Python class that inherits the base class named "ARCGENTest", that is the responsible for performing the tests with the inputs of the child class.

The steps performed by the ARCGENTest class are:

1. Clean-up possible data from previous executions

2. Prepare test environment

3. Perform checks on given ARCGEN configuration file

4. Run ARCGEN with the configuration file

5. In case of a PDS4 test, create the transfer file

6. Generate the report from the given meta-kernel

7. Check that "gen" and "val" folders have the same contents

8. Clean-up execution data

An HTML test report is generated after every test in the 'reports' directory.

Tests performances

Here we show a screenshot with the results of executing the regression tests. Note that the tests are run in about 1,5 secs.